Performance Measurements of Supercomputing and Cloud Storage Solutions

摘要

Increasing amounts of data from varied sources, particularly in the fields ofmachine learning and graph analytics, are causing storage requirements to growrapidly. A variety of technologies exist for storing and sharing these data,ranging from parallel file systems used by supercomputers to distributed blockstorage systems found in clouds. Relatively few comparative measurements existto inform decisions about which storage systems are best suited for particulartasks. This work provides these measurements for two of the most popularstorage technologies: Lustre and Amazon S3. Lustre is an open-source, highperformance, parallel file system used by many of the largest supercomputers inthe world. Amazon's Simple Storage Service, or S3, is part of the Amazon WebServices offering, and offers a scalable, distributed option to store andretrieve data from anywhere on the Internet. Parallel processing is essentialfor achieving high performance on modern storage systems. The performance testsused span the gamut of parallel I/O scenarios, ranging from single-client,single-node Amazon S3 and Lustre performance to a large-scale, multi-clienttest designed to demonstrate the capabilities of a modern storage applianceunder heavy load. These results show that, when parallel I/O is used correctly(i.e., many simultaneous read or write processes), full network bandwidthperformance is achievable and ranged from 10 gigabits/s over a 10 GigE S3connection to 0.35 terabits/s using Lustre on a 1200 port 10 GigE switch. Theseresults demonstrate that S3 is well-suited to sharing vast quantities of dataover the Internet, while Lustre is well-suited to processing large quantitiesof data locally.